Field of the Invention
The present invention relates to a technique for providing mixed reality.
Description of the Related Art
In recent years, as a technique for seamlessly merging a physical world and virtual world in realtime, a mixed reality (so-called MR) technique is known. As one MR technique, the following technique is known. That is, using a video see-through HMD, an object that approximately matches an object observed from the pupil position of a user who wears the HMD is captured by a video camera or the like. An image generated by superimposing a computer graphics (to be abbreviated as CG hereinafter) image on that captured image is presented to the user who wears the HMD.
FIG. 11 is a block diagram showing the functional arrangement of a general mixed reality system (to be referred to as an MR system hereinafter) using a video see-through HMD. An overview of the general MR system will be described below with reference to FIG. 11.
Reference numeral 1101 denotes a video see-through HMD. The HMD 1101 comprises an image capturing unit 1103, display unit 1104, three-dimensional (3D) position and orientation sensor 1105, and I/F (interface) 1106.
Reference numeral 1102 denotes an image processing apparatus which generates a virtual space image (CG) based on 3D position and orientation information generated from captured image information and sensor information received from the HMD 1101, and executes composition processing with a captured image received from the HMD 1101. The image processing apparatus 1102 generally comprises an apparatus such as a PC (personal computer), workstation, or the like, which has an advanced arithmetic processing function and graphics display function. The image processing apparatus 1102 comprises an I/F 1107, position and orientation information generation unit 1108, CG rendering and compositing unit 1110, and contents DB (database) 1109.
The operations of the respective units of the HMD 1101 will be described first.
The image capturing unit 1103 captures an image of an external world at a position and orientation that approximately match the viewpoint of a user who wears the HMD 1101 on the head. The image capturing unit 1103 comprises image capturing elements for the right and left eyes, an optical system, and a DSP (digital signal processor) for image processing of the subsequent stage.
The display unit 1104 is used to display MR images for the right and left eyes output from the image processing apparatus 1102. Therefore, the display unit 1104 comprises display devices for the right and left eyes, and an optical system. As each display device, a compact liquid crystal display or retina-scanning type device based on MEMS (Micro Electro-Mechanical Systems) is used.
The 3D position and orientation sensor 1105 is used to measure its position and orientation. As the 3D position and orientation sensor 1105, a magnetic sensor or gyro sensor (acceleration, angular velocity) is used.
The I/F 1106 is used to make data communications with the image processing apparatus 1102. Images captured by the image capturing unit 1103 and position and orientation information measured by the 3D position and orientation sensor 1105 are transmitted to the image processing apparatus 1102 via this I/F 1106. MR images transmitted from the image processing apparatus 1102 are received via this I/F 1106. As the I/F 1106, a metal line of USB or IEEE1394 or an optical fiber of GigabitEthernet or the like, which can meet realtimeness and can transfer large-capacity data, is used.
The operations of the respective units of the image processing apparatus 1102 will be described below.
The I/F 1107 is used to make data communications with the HMD 1101, and MR images generated by the image processing apparatus 1102 are transmitted to the HMD 1101 via this I/F 1107. Captured images and position and orientation information transmitted from the HMD 1101 are received via this I/F 1107.
The position and orientation information generation unit 1108 generates position and orientation information indicating the position and orientation of the eyes of the user who wears the HMD 1101 based on the captured image information and position and orientation information transmitted from the HMD 1101. As another method of generating the position and orientation information indicating the position and orientation of the eyes of the user, for example, a method using images captured by the image capturing unit 1103 is available.
The contents DB 1109 saves data associated with virtual objects that configure a virtual space.
The CG rendering and compositing unit 1110 builds a virtual space using the data saved in the contents DB 1109, and generates, as virtual space images, images which are obtained by viewing the built virtual space from the viewpoint having the position and orientation indicated by the position and orientation information generated by the position and orientation information generation unit 1108. Then, the unit 1110 generates MR images by compositing the generated virtual space images on the captured images received from the HMD 1101 via the I/F 1107. The generated MR images are transmitted to the HMD 1101 via the I/F 1107.
By the processing based on the aforementioned arrangement, an MR world that seamlessly merges the physical and virtual worlds in real time can be provided to the user who wears the HMD 1101 on the head.
A general system which makes wireless communications between the image processing apparatus and HMD (wireless MR system) will be described below. The wireless MR system can also be implemented by adopting wireless communications between the I/Fs 1106 and 1107 without changing the arrangements of other functional blocks in the arrangement shown in FIG. 11. However, a wireless communication method normally suffers a problem of a considerably narrow transmission bandwidth compared to a wired communication method. Therefore, in order to implement the wireless MR system, for example, an arrangement shown in FIG. 9 is preferably adopted.
FIG. 9 is a block diagram showing the functional arrangement of a general wireless MR system. The same reference numerals in FIG. 9 denote the same parts as in FIG. 11, and a description thereof will not be repeated. An overview of the general wireless MR system will be described below with reference to FIG. 9.
An I/F 1206 is used to transmit position and orientation information calculated by the position and orientation information generation unit 1108 to an image processing apparatus 1202 via a wireless communication.
An I/F 1207 is used to receive the position and orientation information transmitted from an HMD 1201 via a wireless communication, and to output the received information to a CG rendering unit 1210.
The CG rendering unit 1210 builds a virtual space using the data saved in the contents DB 1109, and generates, as virtual space images, images which are obtained by viewing the built virtual space from the viewpoint having the position and orientation indicated by the position and orientation information received by the I/F 1207. The unit 1210 transmits the generated virtual space images to the HMD 1201 via the I/F 1207.
That is, the HMD 1201 does not transmit any captured images to the image processing apparatus 1202 but it transmits only position and orientation information. The image processing apparatus 1202 transmits the virtual space images generated based on that position and orientation information to the HMD 1201. In this way, the data transmission volume exchanged between the HMD 1201 and image processing apparatus 1202 can be reduced.
Upon reception of the virtual space images transmitted from the image processing apparatus 1202, the I/F 1206 outputs these images to an image compositing unit 1211.
The image compositing unit 1211 generates composite images by superimposing the virtual space images received from the I/F 1206 on captured images from the image capturing unit 1103, and outputs them to the display unit 1104.
In the MR system using the video see-through HMD, it is important to assure the visual field of the user who wears the HMD. Particularly, in the wireless MR system that adopts the wireless communication method in transmission of images, the frequency of occurrence of errors increases depending on environments of use and distances between the apparatuses. In addition, switching of image processing apparatuses as communication partners frequently occurs upon movement of the user who wears the HMD, and CG images ceases to be received when another user who is experiencing the MR space intercepts the view with the image processing apparatus as a communication partner.
As a processing method when the communication state becomes unstable due to another user and the apparatuses, patent reference 1 discloses a method of switching a wireless base station as a communication partner based on the radio field strength. According to patent reference 1, when a mobile station detects interference information at a predetermined radio field strength or more, it sends a communication channel switching request to the wireless base station as a communication partner, and the wireless base station sends a new communication channel instruction to the mobile station. When the wireless base station as a communication partner cannot assure a new communication channel, an adjacent wireless base station assures a new communication channel via an exchange. As a channel used to send the switching request and new communication channel instruction, an information channel of a frequency different from that of the communication channel is used.
Patent reference 2 discloses a method of executing channel connection, switching control, and the like by detecting the moving direction and moving velocity of a mobile station.
According to patent reference 2, the moving direction of a mobile station is detected, and when the moving direction is headed toward a position outside a mobile communication service area, a base station notifies the mobile station of the absence of a communication area in the moving direction before the mobile station moves to a position outside the mobile communication service area.
According to patent reference 3, a Doppler shift detection circuit equipped in a radio device of each wireless base station equipment detects the center frequency drift of a channel used by a mobile station, and also detects the traveling direction and velocity of the mobile station, thus automatically executing channel connection and disconnection control.
[Patent Reference 1] Japanese Patent Laid-Open No. 6-69862
[Patent Reference 2] Japanese Patent Laid-Open No. 6-86353
[Patent Reference 3] Japanese Patent Laid-Open No. 2004-15518
However, the aforementioned related arts suffer the following problems.
The technique disclosed in patent reference 1, that is, the method of executing switching control by detecting interference information based on the radio field strength measurement requires the mobile station to always measure the radio field strength. Hence, the consumption power of the mobile station side is large, thus shortening the continuous battery-driven time. Furthermore, since control is done after detection of interference, the mobile station suffers the influence of the interference until switching is completed. Furthermore, when a plurality of users shares the wireless MR system, different images need to be transmitted to the respective users, and a plurality of wireless devices are used within a limited space. Therefore, the frequency bands used by respective HMDs need to be assigned without causing any interference. Therefore, the technique disclosed in patent reference 1 premised on detection of interference information cannot be applied to the wireless MR system.
The techniques disclosed in patent references 2 and 3, that is, the methods of notifying outside the communication range and switching of a base station based on the moving direction of a mobile station do not consider movements of other users and apparatuses. For this reason, these methods cannot detect occurrence of interception of communications due to interception of the view to the base station as a communication partner.